Climbing the ladder of abstraction in NLP

• Bag of Words
  • text is represented as a collection of individual words without considering order or context
• Word embeddings
  • dense, fixed-size vectors capturing word semantics
  • e.g. Word2Vec, GloVe, FastText
• Today: Large Language Models (LLMs)
  • context-aware embeddings using attention mechanisms
  • e.g. BERT, GPT, Llama & Co

Neural Networks

• Computational model inspired by the way biological neural networks in the human brain process information
• Designed to recognize patterns, make predictions, or classify data by mimicking the interconnected structure of neurons in the brain

A simple one-layered NN

Deep Learning

• Oftentimes deep learning & NNs used interchangeably

• Deep refers to the number of layers in the network (i.e. the depth)

• Weights: values that define the strength of the connection between two neurons in adjacent layers of a neural network

• Biases: parameters that allow the network to shift the activation function up or down

• Activation Function: decides whether a neuron should “fire” (pass its signal forward) and how strong that signal should be

  • necessary to introduce non-linearity into the network and handle complex patterns in the data

Transfer learning & fine-tuning

• LLMs are trained on large corpora and thus contain a lot of knowledge about the (written) world
• This knowledge can be transferred to other similar and related tasks
• Transfer learning adds a new layer of neural network on top of the pre-trained model

What is GPU?

• Graphics Processing Unit:
  • a specialized processor to render images and video
  • parallel processing capabilities
  • complements the CPU (Central Processing Unit), which excels at sequential tasks
  • optimized for matrix and vector operations; crucial for neural network computations
• CUDA: Compute Unified Device Architecture
  • a parallel computing platform and application programming interface model created by NVIDIA
  • allows software developers to use a CUDA-enabled GPU for general purpose processing

BERT

• Bidirectional Encoder Representations from Transformers
  • masked language model for NLP tasks
  • only uses encoder part of the architecture
  • bi-directional attention, meaning it looks at the context from both the left and right sides of a word
• RoBERTa: Removed NSP and trained on larger datasets with more robust techniques
• DistilBERT: Smaller and faster version of BERT
• and other variants trained on different languages, tasks, etc

BERT vs. GPT

Issues with LLMs

• Carbon footprint
  • on quantifying ML’s carbon footprint see Lacoste et al. (2019)
• LLMs hardly solve the problem of bias in AI
• GPU dependent
• Data privacy:
  • without personal GPU, you will have to think a lot about your data and whether it ought to be protected
  • some corpora (e.g. copyrighted data) should not be seen by third parties (Google Colab)